Deep Learning has become a popular area, with deep neural network methods obtaining state-of-the-art results on applications in computer vision, natural language processing, and reinforcement learning. This course is well rounded in terms of concepts and helps you understand the fundamentals of Deep Learning. The course starts off gradually with supervised learning and it progresses into the more complicated concepts such as Variational Auto Encoders and Generative Adversarial Networks. We get a complete hands on with Tensorflow/Keras which is very important to implement Deep Learning models. As a student, you will learn the tools required for building Deep Learning models (if you attend the lab sessions!).

Here you will find administrative information for the Fall 1402.

Instructor: Dr. Mohsen Afsharchi, afsharchi at znu.ac.ir

Teaching Assistants: Amin Najafgholi Zadeh, amin.ngz at sharif.edu , Sanaz Khalili, khalili.sanaz94 at gmail.com

Lectures: 8-10 Sat and 8-10 Wed

Prerequisites: Clear understanding of calculus, vector algebra and probability.

The main books for:

1. “Dive into Deep Learning”  By Aston Zhang, Zack Lipton, Mu Li, and Alex J. Smola, 2023.

2. “Understanding Deep Learning”  By Simon J.D. Prince, 2023.

3. “Deep Learning”  By Goodfellow, Ian, Yoshua Bengio, and Aaron Courville, 2016.

Lecture materials will be drawn from the recent online materials and is mainly based on lectures of Dr. Andrew Ng

1. Introduction to Deep Learning (Introduction & History) (Overview of Deep Learning Applications)
   A brief history of artificial neural networks and Deep Learning
   Reading Materials:
   Deep Learning most amazing applications
2. Machine Learning Recap (The Learning Process) (Logistic Regression with a Neural Network Mindset)
   Logistic Regression, Optimization and Gradient Descent
   Reading Materials:
   Logistic Regression
   Binary Classification notes
3. Neural Networks (Shallow Neural Networks) (Deep Neural Networks)
   Reading Materials:
   Neural Networks Notes 1
   Neural Networks Notes 2
   Using Neural Nets to Recognize Handwritten Digits
4. Introduction to ConvNets (Convolution) (Padding, Strided Convolution, Pooling and Convolution over Volume)
   Reading Materials:
   ConvNets: A toy example (Brandon Rohrer)
   ConvNet Notes
   
5. Classic Networks (LeNet, AlexNet, VGG, ResNet, Inception)
   Reading Materials:
   LeNet Gradient-Based Learning Applied to Document
   ImageNet Classification with Deep Convolutional Neural Networks
   VGG-16 Very Deep Convolutional Networks for Large-Scale Image Recognition
   ResNet Deep Residual Learning for Image Recognition
   Going deeper with convolutions
   Taste of Code: SimpleConvNet , Resnet
   
6. A Brief Introduction to Computer Vision (Object Localization) (Multiple Object Detection)
   Reading Materials:
   You Look Only Once: Unified, Real-Time Object Detection
   OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
   
7. Improving Deep Neural Networks (Regularization) (Batch Normalization) (Weight Initialization) (Demo)
   Reading Materials:
   Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
   Initializing Neural Networks
   Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
   Understanding the difficulty of training deep feedforward neural networks
8. Optimization Algorithms (MiniBatch SGD, Gradient descent with momentum, AdaGrad, RMSProp, ADAM)
   Reading Materials:
   10 Gradient Descent Optimization Algorithms
   Optimization Algorithms : Supplementary Code
   
9. Sequence Models (Recurrent Neural Networks)(Gated Recurrent Unit and LSTM)
   Reading Materials:
   Recurrent Neural Networks Tutorial
   Backpropagation Through Time and Vanishing Gradients
   GRU/LSTM
   
10. Word Embedding (Word2Vec)
    Reading Materials:
    Demystifying Neural Network in Skip-Gram Language Modeling
    Optimize Computational Efficiency of Skip-Gram with Negative Sampling
    The backpropagation algorithm for Word2Vec
    
11. Deep NLP Applications (Language modelling, Machine Translation)
    Reading Materials:
    Sequence to Sequence Learning with Neural Networks
    Recurrent neural network based language model
    
12. Transformers (Self Attention Mechanism and Transformers)
    Reading Materials:
    Attention Is All You Need
    The Illustrated Transformer
    End-to-End Object Detection with Transformers
    
13. Large Language Models (Generative Pre-training and ChatGPT)
    Reading Materials:
    Improving Language Understanding by Generative Pre-Training
    Training language models to follow instructions with human feedback
    
14. Semi-supervised Learning (AutoEncoders) (Variational AutoEncoders)
    Reading Materials:
    Variational Autoencoder: Intuition and Implementation
    What is a variational autoencoder?
    
15. Generative Adversarial Networks (GAN)
    Reading Materials:
    Only Numpy: Implementing GAN with Numpy
    How to Develop a 1D Generative Adversarial Network From Scratch in Keras
    

Course Work and Evaluation

Students will be evaluated in this class through a series of exams, an in-class presentation and a semester project. The semester project will involve implementing a non-trivial  program that has been designed using knowledge and techniques covered in class. In-class students will work in teams to work on the project.